Developing effective test sets and metrics for evaluating automated media analysis systems

This paper first looks at current methods of evaluating automated content-based media analysis systems. Several key deficiencies are identified, particularly with regard to test set creation and metric design. A new framework is proposed that better reflects real-world conditions and end-user requirements. This is based on the author’s experience as a professional filmmaker and researcher in this domain. Specific approaches for data set selection, including the importance of understanding the physical, production and aesthetic attributes of footage, are presented. A discussion of related evaluation methods and means of effective assessment follow. It is hoped the suggestions proposed will facilitate more effective analysis of these systems.